Computer and Modernization ›› 2010, Vol. 1 ›› Issue (10): 23-28.doi: 10.3969/j.issn.1006-2475.2010.10.007

• 人工智能 • Previous Articles     Next Articles

Design and Implementation of Job Scheduling Algorithm for Multi-User MapReduce Clusters

WANG Kai, WU Quan-yuan, YANG Shu-qiang   

  1. School of Computer Science, National University of Defense Technology, Changsha 410073, China
  • Received:2010-08-16 Revised:1900-01-01 Online:2010-10-21 Published:2010-10-21

Abstract: As more enterprises start to use data-intensive cluster computing systems such as Hadoop and Dryad for more applications, sharing MapReduce clusters among multiple users that reducing the cost of establishing an independent cluster and the demand of sharing common data sets resources for users is increasing. Based on fair scheduling algorithm, combining with slot allocation delay and priority technology, the paper proposes an improved algorithm. It can achieve better data locality, improve the performance of the system, such as throughput, response time. To meet the differentiated business services, it sets the appropriate for users to ensure special tasks.

Key words: fair schedule, wait schedule, MapReduce, Hadoop

CLC Number: